Multi-Cloud Data Streaming

1. Introduction

b412875666fceda8.png

Last Updated: 2022-02-08

Why the need for Multi-Cloud Streaming?

Businesses are actively looking to spread their workloads across multiple cloud platforms as there are several advantages of adopting multiple cloud providers into a business's operational workflow like:

  • Vendor lock-in
  • Mitigate Business Continuity Risk
  • Dependencies and Reliance on one Technology Stack

What you'll build

In this codelab, you're going to set-up a multi-cloud data streaming between AWS and GCP leveraging Confluent Cloud. You will:

  • Set-up a Dedicated Kafka Cluster in Confluent Cloud on GCP.
  • Set-up a VPC peering between Confluent Cloud and your GCP Project so that it can be accessed on a private network.
  • Setup a VPN tunnel between your AWS project and GCP project so the resources on both hyperscalers can connect.
  • Access Confluent Kafka from AWS and set-up a streaming pipeline from AWS to GCP leveraging Confluent Kafka Cloud.

What you'll learn

  • How to set-up VPC peering between Confluent Cloud Kafka and your GCP project.
  • How to set-up a VPN tunnel between AWS and GCP projects.
  • How to access GCP hosted Confluent Cloud from your AWS project.

This codelab is focused on building multi-cloud data streaming. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

What you'll need

  • AWS project access.
  • GCP project access.
  • Experience of GCP & AWS Cloud.
  • Confluent Kafka Subscription from GCP Marketplace.

2. Getting set up

Setup the VPN connection between AWS and GCP

  • This demo uses the default VPC (asia-southeast1) in GCP and ng-vpc-103-mum (ap-south1) in AWS.

eac371e8af07fc0c.png

c76e9c670e4f8cca.png

  • On AWS, Create the Customer Gateway and the Virtual Private Gateway (VPG). Attach the VPG to the AWS VPC.

55a075a6b951b5df.png

fe23f4315e9369ec.png

  • Create the site to site VPN connection on AWS.

4c59066457f3ff9.png a2dde7c3e7cf82e9.png

  • Download the configuration file for the tunnel created in AWS. Select
  • Vendor : Generic
  • Platform : Generic
  • Software : Vendor Agnostic
  • Ike Version : Ikev2
  • Create the VPN Gateway and the VPN Tunnel on GCP.

8d167a30913a46e1.png

Provide the IP address and the Ikev2 key from the downloaded AWS Configuration file for both the tunnels.

  • Once complete, the tunnel should be UP and Running on AWS and GCP.

baca51d151d7c40.png

ceefee82994ef2d4.png

The tunnel setup is complete now.

  • Select a CIDR block that will be used to configure the Confluent Cloud. Add this to the VPN tunnel as a static route on AWS.

219fa91dc84bd751.png

Confluent Kafka on GCP marketplace

  • Subscribe to Confluent Kafka from GCP marketplace.

380b8221724e91d7.png

  • Login to Confluent cloud and create a Dedicated Cluster

54611bb3ba91a867.png

7e3b9fec07478fc5.png

  • Provide the GCP Project ID along with VPC Network details for peering.

544ea2152de6d447.png

  • Complete the VPC Peering at GCP end with the provided Confluent Cloud cluster network details.

a791c93028f46a10.png

  • Confluent Cloud Cluster is Active and Peered with GCP now.

c9ebd43fab4a8db6.png

  • Test the connectivity to the Confluent Cluster from GCP.
  • Provision a GCE Instance and install Python3.
  • Generate the Python Client Key in Confluent Cloud

969ce69a57fda7db.png

  • Run the test script "producer.py" from GCP instance:

aaede917f462b5b9.png

  • Now, download the example codebase on the AWS instance and execute "producer.py" script to test the hybrid connectivity to Confluent Cloud from GCP.

94da3d8c20f8b234.png

3. Congratulations

Congratulations, you've successfully built your multi-cloud cross-regional Confluent Kafka streaming platform between AWS & GCP.

Helpful Codelabs**?**

Check out some of these codelabs...

Further reading